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FUNCTIONAL LITERACY IN SCHOOLCHILDREN 



DEFINITION AND CRITERIA 
OF TEST SELECTION 



INTRODUCTION ^ ' . 

» ' . 

Pacific Consultants is under contract to Systems 
Development Corporation (SDC) to assist in the selection 
or development of a test of Functional Literacy for use 
irt SDG's Titl^ I Evaluation of Compensatory Education 
Progifams. This report presents the results of the first 
task in Pacific Consultant's work — the definition of • 
functional literacy and criteria fbr evalualiing existing 
tests'. 

GENERAL REQUIREMENTS ' 

An overall description of the characteristics ' 
desired in the test of Functional Literacy vas contained 
in SDC's Statement of Work for the Title I evaluation,^ 
and further clarified in meetings between SDC personnel 
an^ representatives of Pacific Consultants. 

First of all, the ^test must clearly measure functional 
literacy. For the purpose of the Title I evaluation, 
functional literacy is vievfed as the reading atid compu- 
tational skills needed l^y youth as they deal in the 
contemporary non-school related world. It must be an 



independent test designed to measure functional 'literacy 
•rather than the reading and computation portion of an 
achievement test battery.' The level, range, and content 
of the test must be appropriate - for elementary school 
children in grades 4-8, including 'children from disadvan- 
taged backgrounds. 

Cost's of the test should be. in the normal range 
'Of costs for comparable tests. The te^st must be capable 
of group a'dministration by non-expert schoo^ personnel 
employing uniform procedures across the country. 
should be amenable to objective machine scoring pro- 
cedures. / • 
J 

If a norm-referenced test is used, norms appropriate 
to th6 population of the study'should be aviilable: ' -The 
• norms should include appropriate age levels, ethnic groups., 
socioeconomic levels, geographical locations, and language 
considerations. .If. a^criterion reference-^ test is used, 
the criteria on which a test is developed should have 
a valid relation^ to compensatory education goals and 
objectives. * 

Evidence of reliability must be available,* preferably 
a coefficient of at least .80 based on one o£ the generally 
acceptable procedures for deteraiining reliability. The 
computation of reliability should, be based on' data from 
a sample comparable to t^he population of the study. The 



te«t must validly measure performance included in the 
functional literacy goals of compensatory education 
programs. " Evidence of content validity, and/6r validity 
based on comparisons with relevant criteria should be 
available. 

The test must be suitable for the range and type 
of populations in the. study. The- test must be sufficiently 
current to refltect concerns for -the propriety of content 
fqr special groups.. The needs and characteristics of 
participants in the compensatory education programs 
suggest that the pupil population is highly diverse in 
ethnic background. Many pupils are below expected grade 
level in reaching and computation, and some are consid- 
erably below grade level. ' Some pupils speak very little 
English. Nevertheless, the test should measure functional 
literacy in English because the ability -to function in 
American society retquires the^performatice of many tasks 

based on materials witlj- English content. However, a 

■ , ✓ • 

Spanish-language version of the test should be available. 

Fin^illy:, the test should be effective in ineeting 
-the 4najor purposes of the Title I study, which consist 
'of prograon evaluation and comprehensive research concerning 
compensatory education./ The test should be sensitive to 
the effects of major variables , -and sufficiently grounded 
in theory to permit hypotheses to be generated and 
findings to be interpreted in a meaningful fashion. 



III. ASPECTS OF FUNCTIONAL LITERACY 

Social indices of literacy have ysuilly been based ' 
• on years of schooling,, or grade equivalent scores derived 
from standardi;;ed reading -achievement? tests . Assessments 
of .literacy based on such indices hive proved defective 
in two major .ways. First, . they: apparently have Ted to 
a serious underestimation of the iJroblem of literacy in 
the United States, and second, they^have fa^i;ed- to provide 
a meaningful description of the actual capabilities of 
the iJopulation in terms" suitable, as a. basis for policy. 
The defects of such indices have been summarized , else- . 
where (Harman, 1970; Bormuth 1973 ; Naf ziger et. al., 1975) 
'and need not be reiterated here at length. . 

In recent years", an increasing emphasis has been 
-placed on determining -the actual abilities of adults 
performing practical reading tasks involved in reai-world 
life situations, particularly those having social and 
economic utility. Harris (1970, 1971) surveyed adult- 
performance in/ filling out^ common application forms, 
and answering questions based on newspaper employment 
adverj:isements. The National Assessment of Educational 
Progress (1971) conducted a nation-wide sur^vey of adult 
performance on a number of reading tasks bksed on practical 
materials-. The Army has successfully devised measures 
•of reading, listening, and computational capabilities ' 
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required. in specific military occupations based on 
job-related written materials (Sticht, et. al,, 1972). 

Tests designed specifically to measure adult func- 
tional literacy have been developed recently. In the 
course of developing such tests, very substantial prdgr^^ss 
has been made in the conceptualization of functionay 
literacy, and in the methodolbgy^ of literacy assessment. 

The work conducted by Education Testing Service 
under support of the Office of Education's Targeted Research 
ind Development Reading Prograin (now in the National Institute 
of Education) has established a solid body' of data on 
the reading activities of American adults (Murphy , ^1973) . 
A survey based on a national probability sample identified, 
the materials commonly read during ^everyday general 
activities, the duration of reading, and the perceived 
importance of the reading activity. Based on the survey, 
a large number of performance tasks were developed. Each 
task was classified- by type of material, and type of 
socioeconomic benefit (or function) provided by the tasks. 
An advisory panel evaluated the importance of the t^sks, 
and a national survey of adult performance was conducted 
on 170 selected tasks. The data of the survey was later 
used to estimate the economic value of functional literacy 
skills (Murphy, 1975) . 

A domain-r^erenced test of functional literacy (R/EAL) 
was developed by Lichtman (1974) . Through logical analysis" 



she identified nine common classes of reading materials 
representative of everyday life activities and "selected 
one specific* type of material within each, class. A 
specific behavioral objective was written fir each typo 
of material, and a task analysis conducted to identify 
subordinate objectives. Items'^were then written to 
cover the tepninal and subordinate tasks in each hierarchy, 
using realistic facsimiles of actual sample materials. 

To date, no efforts 'to develop functional literacy 
•measures for schoolchildren, i^i grades 4-8 have come to 
light. However, the previous wprk on adult functional 
literacy cle^ly indicates severaj. primary dimensions 
which must bfe consfdered in> defining the .concept for • 
school-age children, and whioh must be operationalized ' . 
;in the process of ^lest development. These dimensions " ,' 
include: . • . 

1. The area of everyday life activity in which a""" 
functional literacy task is encountered. 

2. The ^equency and duration of encounters with' 
the task. | 

3^ The type of material involved in the task. 

4. The specific behaviors required by the task. 

5. The type of social function 'served by the task. 

6. The importance of the t^sk, as perceived by 
the individual- ^taking tihe test.; 
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7. The degree of socioeconomic benefit derived from 
successful performance, measured in e<fonomic 
t€irms, or as perceived by significant authorities. 

One further dimension of the functional literacy 
domain has been suggested by Borjnuth (1973). This is 
the linguistic characterization of the corpus of language 
associated with the materials. No serious consideration 
has as yet been given to linguistic factors in functional 
literacy, aside from identifying the mother-tongue used 
(e.g. , English) . 

DEFINITION OF FUNCTIONAL LITERACY 

No definition- of functional li^teracy has yet been • . 

generally acceptec^.» Bormuth| has suggested an ajl-inclusive 

definition of literacy in the followirSg terms: 

"In the broadest sense o^ the word, literacy ^ 
is the ability to exhibit all of the behaviors 
a person needs in order to respond appropriately 
to cill possible reading tasks." 

Nafziger, et. al. (1975) point oiit that literacy, 

unlike reading, refers both to basic reading skills and 

to socially appropriate reading behavior. They go on to 

indicate that functional literacy implies reading for a 

purpose, and a purpose related in some way to social 

utility. 

The U.S. Office ofy^ducation has been quoted by 
Nafziger et. al. (1975) as defining a literate persor^ 
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in the following terms: ^ • 

".•.one who has acquired the essential knowlodqe 
and skills in reading, writin<|i, and computatVion 
required for effective functioning in society, 
and whose attainment makes it possible for him 
to develop new aptitudes and to participate actively 
in the life of his times." 

None of these definitions offer very sp.ecific guidanc^e 
to the operationalization of the functional literacy 
concept as a basis for development of an effective assess- 
ment device. For ^the purpose of the Title I evaluation 
of Compensatory Education Programs, a definition is 
offered h^re which' Ifends itself to operationalization 
in terms of the dimensions of functional literacy iden« 
tified in the previous section*. 

Functional literacy of school children in the' 4-8v 
grade range is the capability of perf orminor^in a consis- 
tently successful manner t.hose reading and^ computational 
tasks which: i 

1. are normally encountered in the course of 
everyday life activities by a majority of 
children in non-school settings. 

2. are normally encountered repeatedly, or involve 
^ a substantial duration of activity. 

3. involve commerce with particular types of 
materials commonly found. in the environment of 
the child. 



4. involve a specific corpus of rion-techni^al j 

language and symbolic representation associated 
X with the materials. 

5'. Requf're specific observable behavior in relation 
to the material. 

6. serve definable types of social functions. 

7. are regarded as important by the child or 
established authority figures, or have demonstrabl 
and non-trivial socioeconomic benefits. 

This definition is not fully operational , .but can"" 

easily become so given the specification of standards 

for qualifiers and the drawing up of a number of systems ^ 

.of classification covering the dimensions mentioned in 

the definition. Suggestions for the operationalization 

of each aspect of the definition are presented in the 

' following section. 

» 

REQUIREMENTS FOR OPERATIONALIZATION «- 

r • ■ 

Lacking a survey of reading activities for children 
in the 4-8 grade range comparable to that conducted with 
adults by ETS, it IS relatively difficult to fully opera- 
tionalize any definition. of functional literacy. Neverthele 
a considerable amount of progress toward this goal can be 
accomplished through ^ logical analysis, the advice of experts 



Tabic I 

Areas of Life Activity* for Children 



Personal Maintenance 

a. Food 

b. Clothinjf 

c. ^ Health " 

Personal Relations 

a» Family 

b. Friends 

c. Relatives 

d. Pets 

Institutional Relations 

a^ School 

b. Church 

c. ^Club 

d. Police and Fir^ 
Neighborhood Locations 



5. Home Maintenance 



1 



> 




a. 


Home 


b* 


Shopping 


c. 


Postal 


d. 


School 


e. 


Medical 


f . 


Recreation 


g. 


Library 



a . Appliances ^ 

b. Yard and Garden 

c. Furniture 

d. Radio, . V . , Sj^ereo 



6. .Travel 



a. 


Bicycle 




Bus 


c. 


Car 


7. Leisure 


a. 


Sports 


b. 


Games 


o. 


Toys 


8 . Communications 


a . 


Newspapers 


b. 


Comics 


c . 


Television 


d. 


Books 


e . 


Movies 


f . 


Mail 


g- 


Telephone- 



and by capitalizing on certain aspects of previous work 
done with, adults. 

The first major point of the definition proposed 
here is that tasks should be "normally encountered in 
the course of everyday life activities by a majority ^oT • 
children in non-school settings." This aspect can be * 
operationalized in either one or both of two ways: 

1. Given a comprehensive list of everyday life 
activities, SDC's literacy panel can be asked 
to nominate represew-tative tasks which Iri their 
best judgement are "normally encountered by a 
majority of children." 

2. Based on a comprehensive list of everyday life 
activities, a survey can be taken on a small 

but -diverse sample df children who would nominate 
tasks which the^ could remember having pei^formed 
in each area of life. 



As a basis for thi55' step, a tentative ^j^^^ ^ life 
activities is presented in Table I. In connection with 
each area of life, several significant entities are listed 
which might servo as effective stimuli-arousing asnociationn 
to a reading or computation task. For example, a child 
asked to remember what ho haf> computed relative to mom or 
dad, might remember havi ng calculated one of the parent's 
ages, given the dates of birth. Or, asked about reading 
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y Table II 

Types of Materials 
* ■ 



1. 


"Signs, Labels 


2. 


Schedules, Tables 


3. 


Maps, Diagrams 


4/ 


Categorized Listinac; 


5. 


Directions, Instructions 


6. 


Advertisements, Announcements 


7. 


Forms ^ 


8. 


Personal Communications 


9. 


Instruments, Controls 


10. 


Technical Documents 


11. 


Discourse, Narrative 



T 



/ 
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in connection with a post office or mail box/ the child 

might remember reading the i^il pickup times, and such 
♦ 

labels as "local," "out-6'f-town, " "air maif^,." "A.M.," 
"P.M. , " etc. . . 

If a survey of children is performed, ^ata can also 
be gathered on the frequency and duration of ^ each task, 
which can then be used to form basic standards for the 
second point in t^e definition. Otherwise, the question 
of frequency and duration will have to be left to expert 
judgement. ^ 

The third point has to do with the specif :^;,rfHi^erials 



associated with a given t^sK. The SDO literacy danel 
or the surveyed children 'can be asked to identify the 
materials used in connection with each task. However, 
it is also useful to have available a systematic class-, 
ification system of materials, both as a stimulus to/ 
memory, and as a refinement and explicatibh of the third 
point in the definition. Such a classification sy$tem 
is presented in Table II. The table has been^ based 
largely on the classification of materials used by ETS 
(Murphy, 1973) and Lichtman (1974). 

There is presently^no basis for explicitly defininq 
the corpus of language and symbolic repr:csentations 
involved in children's functional literacy. Study of 
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the riiessages conveyed by specific materials included 
in an extensive sampling of materials would be required 
to empirically establish such a definition. Some 
restrictions might be proposed based on consultation 
with experts in ch^d lan^Aaage development. Clearly, ' 
a child should not be expected to read language and • 
handle concepts that are hot^et incorporated into his 
spoken language competence, or that lie outside his 
normal realm of experience. One such restriction/ con- 
fining the corpus to non-technical language has been 
included in the definition. ^Language should be considered 
technical if it occurs la::gely within k narrow field 
of social activity, is usually known primarily by persons 
directly engaged in that activity, and if ,nost children 
are unfamiliar with the worfl, expression/ or syntactic 
construction in question. . ' ' 

In the area of computation, similar restrictions 
can be proposed based on the normal course of school 
instruction.. Cle'arly, no advanced forms of computation 
or symbols which children have not had an opportunity 
to learn should be included. ' 

At this point in time, it is also rather difficult 
to propose a system of behavior categories which can be 
expected to provide a valid delineation of the specific 
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behaviors/ involved in functional literacy tasks. To a 
large extetit, the nature of the reading tasks seem 
implic/t in, and inferrable from, the material associated"^ 
with the tasks. For exaunple, the typical behavior 
invoaved in using a. telephone directory is describable 
iiyterms of a systematic search algorithm^ given a 
specific entry neune, proceeding through subgOals defined 
Dy a sequence of alphabetic cues, and eveVvtuating in 
location and retrieval of a specific numeric eode from 

V 

the listings. The usual behaviors associated with filling 
out forms include reading and comprehending headings or 
questions which identify requested information, retrieving 
personal information from memory or available records, 
and writing the information in appropriate blanks or 
checking off appropriate alternatives. 

A systematic task-skills analysis of a considerable 
number of such tasks is required to define a comprehensive 
set of component- skills covering the /domain of functional 
Veading tasks. Lacking siich an analysis, an informal 
list of behavior categories is offered in Table III, 
based on a cur soryl examination of the 170 examples of \ 
adult materials used by ETS. in constructing test items 
(Murphy, 1973) and those, listed by Lichtman (1974). This 
list certainly fails to include some forms of behavior 
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'Table III 



^Preliminary c;.assif ication System-fbr 
Behavior Exhibited in Functional Litei^acy TSsks 



Behavior 



Material 



Systematic Seirch 



Choice Discrimination 



Preference Selection 



Retrieval of Personal Data 

Selection and Storage 
of Information 

PerfoMing Sequence 
of Operations 

Contingency Identification 



Acctaracy Verification 

domprehension of 
Information 



CategtDrical Listings,. 
Advertisements 

Signs, Lab^lsv Maps,. 
Diagrams ^ Instruments / 
Controls 

Categgrjssed Listings, 
Adve r t i fe erne n ts 

Personal * Communication , 
Discourse^ Narrative 

Directions , Insttuctiohs , 
Maps 9 Diagrams 

Signs, Labels, Directions, 
Instructions , Technical 
Documents 

Technical Documents, 
Fortns 

Personal Communication, 
Maps , Diagrams 



P4?tic^l^^ly iinportant for children, and others only 
relevant for adults, but it is a useful basis for later 
revision.' Also listed in the table are the kinds of 
material which are most usually found associated, with 
each behavior category. • . , 

A somewhat different tack can be taken in specifying 
the>^oinputational behavior s.^ .Real-world computational 
f problems can be distinguished on the basis of the . i 
computational operations/ the nvlmericial and measurement 
content, and the meansi of computati^. This provides 
■the basis for a facet design of cdmputational categories 
shown in Table IV. As shown, the child will perform 
confoutational tasks associated witK real-world materials 
requiring operations , based on signed numbers representing 
physical quantities , expressed in units of ■ i^easurement 
by appropriate means. 

ETS identified eight categories of socioeconomic 
functions benefits as shown in Tablq V. These seem to 
overlap to some extent with the categories of life activity 
presented in Table I. Revisions of these categories to 
make them more appropriate for children may be based on ' 
recommendations of SDC's literacy panel or consultation 
with experts in child development , or based on survey 
responses from children. 



Table IV 



Facet Design for 
Computational Categories 



^ Operations . Signs ' Numbers 



Addition ^'Positive Integers 

Subtraction N^gy^itive Fractions 

Multiplication # ^ • DeciiUals* 

Division Percentages 

Quantities Units Means 

I^pngth ' ^ .English , Mental 

Area Metric Paper & Pencil* 

Volume . Kitchen 

Weight 

Time -J 
Dry Measure * 
Liquid Measure 
Speed 
Money 
Cost/Unit 
Angular 



Table Vo 

Functions of Literacy Behavior 



1. Economic 

2;, Occupational^ 

3. Education/ Culture • 

4. Reci'eation ^ 

5. Health^ 

6. Maintenance 

7 . Personal Relationships 
8 • Citizenship 
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- . \ : ■ ■:■ ■ . 

Evaluation o'^^the importance «of tasks can also 
be based -dn jjudge^^nts of the Literacy Panel or survey 
responses. It may also be useful to have teachers and 
parents examine the degree of benefit to be attributed 
^l*to specific tasks, since they bring 'somewf^t different 
but significant perspectives to this issue. 




\ 
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SELECTION CRITERIA 

Characteristics of the test, the nature of the 
examinees, and the purpose of testing ^r'e important 
factors in selecting a test of functional literacy ' 
for use in the Title I Evaluation. The criteria for 
test selection presented here are based very l^ggly 
on the general guidelines provided..by the American 
Psychological Association's standards for Educational 
and Psi^chological Tests, and the criteria employed 
in the test evaluations at the Center ^or the Study 
Of Evaluation (CSE) , as presented in the ^ocument 
CSE Elementary School Test .Evaluations , authored by 
Ralph Hoepfher and cj)thers. Additikial criteria were 
•suggested by Pacific Consultants' previous review of 
reading and literacy tests for the Right-to-Read 
Evaluation, and the recent examination of tests of 
adult functional literacy performed at the Northwest 
Regional Educational Laboratory under the direction of 
Dean Nafziger. 

The criteria suggested by the sources indicated 
above provided a reasonably complete compilation of 
factors relevant to test selection, but were not con- 
cerned specifically with the- measurement of functional- 
literacy in grades 4-8 for the purpose of program 



evaluation. A number of general reconmendations were 
not suitable in meeting the special requirements of 
the Title I Evaluation, and were therefore modified 
as necessary. The proposed criteria are organized 
according to the four general areas of 1) measurement 
validity, 2) examinee appropriateness ,, 3) technical 
quality, and 4) administrative usability in corres- 
pondence with the four areas identified in the CSE 
test 'evaluation system. 

Validity 



The criteria in this area concern the nature of 
what is measured by the test. It is most important 
that the test be clearly and unambiguously a measure 
of functional literacy, if ig. role i^ the Title I 
Evaluation is to be serveSML^ofs contributing to 
the credibility of a tesy;^ft'i«easuring functional 
literacy are considered in terms of content, empirical, 
and construct validity. — 
^' , Content Validity - It ip highly desireable 

that the test be representative of a defineable 
population, of items and performances with 
specific reference to the domain of func- 
tional literacy. The bases of definition and" 
the i>rocedures of test construction contribute 
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to content validity in terms of . the criteria 
outlined below. 

1. Definition - The test should, be speci- 
fically designed as a test of functional 
literacy. Disagreement on the validity 
of content will surely arise if" the test 
was originally designed for some other 
purpose, and if no explicit' basis exists 
for judging the relevance of items. 

2- Material Domain - The stimulus materials 
should be representative of those commonly 
encountered in real-life reading and compu- 
tational tasks. Confidence in the 

i 

representativeness of materials would be 
increased if a population of such materials 
were defined, the composition of the pop- 
ulation was described in terms of types 
or characteristics of materials and formed 
pa^t of the definition of functional 
literacy used as the basis of test devel- 
opment . 

3. Behavior Domain - The performance required 
. ir> the items should be ^Representative of the 
tasks "and ajcills commonly required in real- 
life reading and computational performances.. 



ERIC . . ' -23- 



4 



Explicit classificatlbn and/or description 
of a domain of functional literacy behaviors 
is desireable as part of the definition used 
as a basis for test development. 
■ Symbolic Domain - The language and other 
symbolic represejitations which form the 
communicative component ofj^e materials 
should be representative of the symbolic 
content commonly encountered in real-life 
reading and computational tasks. Specifi- 
cation of the symbolic content in linguistic 
and mathematical terms can further strengthen 
^nd clarify the definition of functional 
literacy beyond the material and behavior 
specifications usually considered. Such 
specifications could be particularly helpful 
in defining levels or ranges of .competence 
in relation to the domains of materials 
an^ tasks. 

Socioeconomic nnm.-in . The materials and 
tasks should be representative of the socio- 
economic functions commonly encountered 
in real-life reading and computational tasks. * 
A classification o;: description of socio- 
ecori^ac functions and the benefits or values 



-24- ^'^ 



of performance should be part of tha-r --^ 
definition of functional literacy used ', 

t 

as a basis of test development, to help 
insure that functionally significant rather 
than trivial performances are represented. 
6- Program Objectivpc - The materials and 

tasks of the functional, literacy test should 
not be referenced to sV^clTrc"' program objoc- 
_.4:ives. Program-referencing would amount 
to prejudging the result of the evaluation 
in relation to functional literacy, in that 
it would inevitably bias the evaluation in 
favor of program goals and those programs 
which emphasised the defined^ objectives. 
The test is intended to provide an objective 
criterion by means of which the effectiveness 
of various programs can be judged in the 
area of functional literacy. 
Item Cons truction - Procedures used in 
constructing the items should be explicitly 
defined anA replicable. Items should not 
permit response on thi basis of mere word- 
matching between stem and answer. Rigorous 
item, construction procedures are required 
to ensure the relevance and representativeness 
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of the item pool in relation to the aspects 
specified in tHe definition of functional 
literacy, whether material, behavioral, 
symbolic, or socioeconomic criteria are 
included. Procedures which are entirely 
algorithmic would be most advantageous but 
are not Within the usual state of the art 
at present. Other procedures are acceptable 
if the resulting items show close correspon- 
dence to the classification systems employed 
in defining functional literacy. 
Item Selection - Procedures used in selecting 
items from a pool for inclusion in the final • 
test should be empirically based, and yield 
evidence that the representativeness of the. . 
pool was maintained or increased in the item 
selection process. ' , ^ 

Subtests - The t,est should incorporate sub- 
tests for reading and computational tasks, 
but which may be related to the same stimulus: 
materials. 

j:riteria l Objectives - The definition of^ 
functional- literacy s^hould be supplemented 
and operationaliVed by the spebiflcation o'f 
a set of criterial tasks referenced directly 



to the characteristics of materials, behavior, 
symbolic content, and functions employed in 
the functional literacy definition. Such 
objectives would provide an important link 
between definition and items. Such objoc- 
tives might be used in item construction 
and selection, ^or as a basis for empirical 
validation of items. 

B. Empirical Validity - it is desireable that 

the test have been useji in' previous studies, „ 
thus providing empirical evidence relating the 
test, scores meaningfully to other variables. 
Areas of concern in relation to .empirical valid- 
ity aj^ outlined below. 

^' Concurrent Relations - It is Advantageous 
but not essential that the test has been 
correlated in previous studies with a wide 
variety, of other measures tak^n at the same 
time. The number and quality of studies, 
the number of variables, and the diversity 
of variables all cftntribute to the evidence 
bearing oh the meaning 'of a given literacy 
score. 

•2. Predictive Relations - It is advantageous 
but not essential that the test have been 
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.correlated with measures taken at some 
later time. The number and quality of 
studies, as well as the number and diversity 
of variabljg^contribute to the evidence 
bearing on the qiiestion of what consequences 
flow from having attained a given literacy 
score. 

Causality - It i^ advantageous, thoUgh not 
essential, that studies have been performed 
which relate the functional literacy test 
to important psychological , educational , 
or socioeconomic independent variables . Such 
evident should- be of assistance in the 
•analysis and interpretation' of the findings 
in the Title I Evaluation. 
Nature of Relat^ns - Empirical relation- 
ships found in the available literature 
should be reasonably interpretable in terms 
of prevailing educational, psychological,' 
and socioeconomic theory. The measure of 
functional literacy should relate sensibly 
to variables which can be isxpected to 
influence functional* literacy, variables 
which can be considered to reflect components 
of functional literacy, and variables whict> 
are thought to be independent of functional 



. literacy. Factor analytic studies, if an/ 
are available, should indicate that. the / 
, measure of functional literacy is facto/ially 
complex. The nature of one particular/ 
relationship is especially important/ fhe 
subtests should not correlate too hl/ghly 
with standardized tests of reading Ability 
or pure computational skills. Ver^- J;igh 
correlations of this sort would indicate ' 
that the test did not adequately represent 
the diversity of skills required in a func- 
tional literacy measure. 
5. Sensitivity - It is advantageous that the 
magnitude of effects observed was substan- 
tial when the test was used as a dependent 
variable in experiments or evaluations. That 
is, the test should be sensitive to the effect 
of appropriate independent variables, so 
that there is some assurance that appro- 
priate effects will be revealed in the Title 
I Evaluation as well. 

Construc t Validity - Criteria in this area have 
to do with the theoretical basis of the functional 
literacy concept. They are of lesser importance 



I ■ .' 

in judging validity than content and empirical 
criteria given the practical concerns of the 
Title I Evaluation, but are valu^le charac- 
teristics nonetheless. 

!• Process Constructs - The conceptualization, 
development, and empirical validation of 
the test should be grounded on relevant 
psychologicall linquistic, educational theory 
in the are^^f reading and computation. 
Particularly important in this respect is 
the availability of a task-skills analysis 
which would define the components of functional 
literacy, indicate hierarchical relations 
among components, and. tie performance 
to basic/ cognitive information processing 
6perations. Such a theoretical foundation 
is useful in generating hypotheses and 
interpreting results. 

2. Acquisition Constructs - The conceptualization^ 
development, and empirical validation of the 
test should be grounded in relevant psychol-^ 
ogical, linguistic, and educational theory * 
in the areas of instruction and cognitive 
and language development. ^Such formulations 
would provide a basis for tying changes 
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in functional literacy to specific educa- 
tional practices, and related developmental 
changes . 

^' Socioecon omic Constructs - The conceptual- 
ization, development and validation of the 
test should be grounded on relevant social 
, and economic theory to provide a basis 
for hypothesis and interpretations of 
findings concerning relevant socioeconomic 
variables, and the function and benefits- 
of lit^l^cy. . 

Appropriateness 

The second set of criteria concern the appropriate 
ness of, the test in relation to characte^ristics of the 
intended sample of examinees. The criteria focus on the 
fcliree areas of instructions, items, and format and 
procedure. 

A. Instructions 

1. ' Clarity - The instructions should be \ 
appropriate in orientation and tone, 
inoffensive in content, and comprehensible 
with vocabulary and syntax suitable for 
children in the 4-8 grade range. 



2. Purpose - The instruction^ should provide 
an honest explanation of its purpose and 
intended use. 

3. Comprehensiveness - The instructions should 
precisely and completely describe all require- 
ments of the tasks presented in the items 

so that the examinee has all the information 
needed to adopt an effective performance 
strategy. 

4. Sample^ Items - The instructions should include 
sample items accurately illustrating- task 
requirements and the level of difficulty 

of the tasks. 

5. Mode - The instructions should be presented 
in an oral mode. If administratively 
feasible, the instructions should be tape 
recorded/*^ or if not, a standardized script 
should be available suitable for fluid oral , 
reading by non-expert examiners. 

Items 

1. Difficulty - Items should include a wide 

range of difficulties, including some items 
relatively easy for 4th grade children, and 
some .items relatively difficult for 8th graders. 



The distribution of dif f iculti^es should 
be soiTlewhat flatter than is usual for an 
achievement test so that the same test 
can be used over the whole 4-8 grade range. 
Thus, approximately equal numbers of items 
with 'moderate levels of difficulty at each 
grade level should be included. 

2. Motivation - The items should be relevant, 
up-to-date, and interesting for Children in 
the. 4-8 grade range so as to arouse intrinsic 
motivation in task performance, without 
extensive exhortations being required to 
induce cooperation anS effort. 

3, Propriety - The content of the items should 

j 

not involve any invasion of pri*vacy, or / 
any sexist, racist, &t otherwise offensive 
aspects of content. 

Format and Procedure y^^^ 

1- Physical Quality - The paper should be of 
good 'quality , the print bold and readable, 
and the illustrations clear and up-to-date. 
Reproduction of mate,^ials involved in common . 
reading and ^computational tasks should present 
realistic facsimiTies of the actual materials, 
preferably including full-color reproductions. 



2. Layout - The test should t>e effectively 
arranged and cued to facilitate, recognition 
of items as unit's, the perception of the 
relation between item stems, answers, and * 

, examinee response, and the progression of 

r 

successive items and pages. 

3. Timing - The test should be time limited ' 
but permit most examinees to attempt most 
items within the time allowed. Sectioning 
of the test, with timing instructions for 
each section may help to maintain appro- 
priate pacing in the brief time alloted 
for this test. Items at all difficulty 
levels should be represented in each section. 

4. Response Mode - The response should be 
marked directly on the test sheet, in a 
^^^^on permitting machine scoring. No 
s^a^ate answer sheet should be required. 

5. Compleyity - Each item should require one 
simple artd direct responsfe, with no multiple 
steps or complications other than those 
intrinsic in the task represented by the 
item. Several items might be used based 

on the same stimulus materials provided 

the relationship of each item to the stimulus 

is clear. 

s 
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Technical Quality 

Criteria addressed in this section pertain to 
the reliability and comparability of the test scores, 
and the quality of noinnative standards. 

A. R'bliability 

1. Comparability - Alternate forms should be 
based on parallel xteifigNwith comparable 
item statistics. The fopris should correlate 
.80 or above at every grade level in the 

4-8 range. • ' 

2. Stability - Test-retest correlations should 

be .80 oar above over brief time intervals, 

■ ' / 

i.e., one month or less. Reliability 
coefficients could b<6: lower oy«r longer 
intervals, particularly when instructional 
experiences have intervened having a sub- 
stantial effect on the level of functional 
literacy performance. However, in the 
case where no shift in level of performance 
has occurred, the reliability should remain 
above .70 for intervals up to one year. 

3. Internal Consistency - High internal con- 
**feistency is not a necessary criljje^pion for 
the functional literacy test. A test which 
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is highly homogeneous is likely to be 
unrepresentative of the full diversity of 
tasks which should be sampled in a functional 
literacy test. iSi partic\ilar, items involving 
reading should only be moderately related to 
Computational items. The. correlation 
between reading and computational subt^ests, 
if present in th^ test, should correlate 
below .70, and preferably below .50. 

Normative .Standards 

1. Data Available - Although normative"' data 
is not essential in view of the large 

sample to be tested in the Title I Evaluation, 
and the emphasis on program comparison in 
the evaluation, it will still be\help,ful 
to have some prior normative data available 
as a basis tor comparison. 

2. Normative Sample - It is desireable that 
normative data be available for the 4-8 
grade range, and for adult^ as well. 

3- Representative - It is de(^ireable that the 
sample be representative of racial, ethnic, 
sex, geographic and socioeconomic strata, 
rather than thlfe result of incidental sampling. v 



4. Reporting - it is desireable that normative 
. data be reported separately as well a.s 

combined over t^e racial, ethnic, geographical, 
and socioeconomic strata represented in the 
- sample.- 

5. Item Statistics - it is useful if item 

, statistics are reported both for the whole 
•sample and^birok.en down by strata. Item 
difficulties are the most " important statistic, 

, • but item discrimination indices and inter- 
correlatiohs are useful as well. 



Usability 



Criteria considered in this area pertain , to the 
practical aspects of administration, scoring, and inter- 
pretation. 

A. Administration ' ( 

1. Personnel - Non-expert school personnel " 
Should be capable «of administering the 

test with very little training. The services 
of a specialist or a testing expert, or 
extensive training should not Be required. 

2. Scheduling - The test should require no more 
than 30 minutes of- t/sting time (prefer^y 



20 minutes) at one occasion of testing. 
Tests taking longer than 30 minutes should 
be easily modifiable for shorter le^igth 
with no more than normally expected loss 
of reliajjility . 

Conditions - The test should be capable 
of admiriisti'ation in usual classroom settings 
,to group sizes in . the- normal range for intact 
cl.dssroom, groups, and without the necessity 
of special equipment (othejr than a cassett 
tape player, if it is -decided to use^aped . 
instructions) . 

- The test should be scored in an 
objective manner by machine. Machine scoring 
should be highly flfdil-safeaHd reliable, 
without complex error checking routines to 
proof the results. 

Components - The test materials should be 
entirely of .the paper-and-pencil test variety, 
with no special manipulanda, slides, or 
other unusual components. 

Cost - Costs should be in the normal range 
of paper-and-pencil tests having good quality' 
paper and printing_, including color repro- 
duction. • ' 
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Interpretation 

1. Manuals - high quality test manual 

should be available, one which meets the 
appropriate APA standards for test manuals. 
A supplemental brochure 'describing the test 
and how to interpret its scores should also 
. be available for relatively unsophisticated 
consumers of the iresults. 
. Meaninc^ - The test scores should be highly 
meaningful and "nderstandabi^in terms of 
'specific performance by a nontechnical 
audience including the general public. it 
would be most meaningful if a hierarchy of 
performance levels could be devised, in 
which a person placed at ,qne level could 
be described as capable of a specific list 
of tasks, and all tasks listed at lower 
levels. However, this may be an unrealistic 
goal. 

^c^'^GS - The primary test scores should 
be directly understandable in absolute torme 
.without the use of complex conversions or 

. v. 

scaling. Forms of scaling or convers'ion 
to standardized scores may bo used hr. a 



supplement to the. ptimary. scores or for 
use by ..audiences with a higher level of 
technical background. 
Implications - It is desireable that 
the implidations of givan test scpres'for 
educational practice "or public policy be 
clear and relatively direct. However, 
what is actually required to meet this 
criterion is not entirely certain. 
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